Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support chDB as a driver #369

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Support chDB as a driver #369

wants to merge 8 commits into from

Conversation

ThomAub
Copy link

@ThomAub ThomAub commented Oct 18, 2024

Summary

This is a first proposition for #297. It's currently working for our use case of embedded ClickHouse for unit tests with DBT.

It would be great to have some feedback in the direction of this feature implementation !
We would also be interested in how should we add Unit and integration tests or even a tutorial ?

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@CLAassistant
Copy link

CLAassistant commented Oct 18, 2024

CLA assistant check
All committers have signed the CLA.

@ThomAub
Copy link
Author

ThomAub commented Oct 18, 2024

also tagging @auxten if you have insight on the chDB part

@auxten
Copy link
Member

auxten commented Oct 21, 2024

It looks great. I'm here for any issue or question :)

chdb-dev:
type: clickhouse
driver: chdb
chdb_state_dir: "chdb_state"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in dbt target provided folder

type: clickhouse
driver: chdb
chdb_state_dir: "chdb_state"
chdb_dump_dir: "dump"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe more granular and able to load single file or glob expression

@auxten
Copy link
Member

auxten commented Oct 28, 2024

@ThomAub There are some lint issues to fix, and we also need unit and integration tests.

You can "Allow edits from maintainers."

@BentsiLeviav
Copy link
Contributor

Hi folks,

Thank you for your contribution!

Before reviewing your PR, please add the following:

  • A short description with a link to this PR in the changelog (please keep the current format we have in the Changelog file).
  • As this feature isn't minor, please add tests to cover the added functionality and documentation to the readme file.

Looking forward to reviewing this!

jpelletiPL and others added 2 commits November 22, 2024 16:35
Added unit test for chDB dbt materialization
Updated changelog with a description of the new feature
feat: Add tests and update changelog
@auxten
Copy link
Member

auxten commented Nov 26, 2024

@ThomAub In the chdb v2.2.0b0, the chdb.dbapi is totally refactored. The API and behavior suppose to be not changed.
The good parts are:

  • Stateful Query with long live clickhouse engine instance bind with connection
  • Python DB-API reimplemented with new API. Both persist and memory engine supported
  • ClickHouse memory engine support
  • Some performance improvement (~43%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants